{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploration vs Exploitation\n",
    "\n",
    "In this notebook we will look at the exploration vs exploitation dilemma. We will consider a simpler environment known as **Mulit Arm Bandit**.\n",
    "\n",
    "We will implement three variations of exploration strategy and compare their relative performance. \n",
    "\n",
    "Our Bandit has $K$ actions. Each action has a probability of success $\\theta_k$ which is known to us. When you pull the kth arm, it produces a reward $r$ of 1 with probability $\\theta_k$ and reward $r$ of 0 with probability $1-\\theta_k$.\n",
    "\n",
    "Let the optimal action be defined as the one which has maximum probability of success i.e. $\\theta_k$.\n",
    "\n",
    "$\\theta^* = \\max_k\\{\\theta_k\\}$\n",
    "\n",
    "The regret at a time step t is defined as $\\theta^* - r_t$\n",
    "\n",
    "The total regret for a fixed time horizon T is defined as:\n",
    "$$\\rho = T . \\theta^* - \\sum_{t=1}^T r_t$$\n",
    "\n",
    "We will use three different exploration strategies and see how total regret grows as a function of T."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bandit Environment\n",
    "`Bandit` implements the environment. We are using non contextual multi arm bandit. In other words the environment has no state, we take an action and observe the reward and the episode terminates. In next time step, we choose another action as per our exploration strategy to get another reward followed by episode termination. This cycle goes on for T time steps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Bandit:\n",
    "    def __init__(self, n_actions=5):\n",
    "        self._probs = np.random.random(n_actions)\n",
    "        self.n_actions = n_actions\n",
    "\n",
    "    def pull(self, action):\n",
    "        if np.random.random() > self._probs[action]:\n",
    "            return 0.0\n",
    "        return 1.0\n",
    "\n",
    "    def optimal_reward(self):\n",
    "        return np.max(self._probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Sampling Agent\n",
    "\n",
    "In this the agent randomly chooses an action from available actions and records the regret. There is no exploitation. Only random exploration. We know that it will behave very badly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RandomAgent:\n",
    "    def __init__(self, n_actions=5):\n",
    "        self.success_cnt = np.zeros(n_actions)\n",
    "        self.failure_cnt = np.zeros(n_actions)\n",
    "        self.total_pulls = 0\n",
    "        self.n_actions = n_actions\n",
    "        \n",
    "    def reset(self):\n",
    "        self.success_cnt = np.zeros(n_actions)\n",
    "        self.failure_cnt = np.zeros(n_actions)\n",
    "        self.total_pulls = 0        \n",
    "        \n",
    "    def get_action(self):\n",
    "        return np.random.randint(0, self.n_actions)\n",
    "    \n",
    "    def update(self, action, reward):\n",
    "        self.total_pulls += 1\n",
    "        self.success_cnt[action] += reward\n",
    "        self.failure_cnt[action] += 1 - reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Epsilon Greedy Sampling Agent\n",
    "\n",
    "This is similar to the $\\epsilon$-greedy strategy we have seen in all previous chapters. The agent tries different actions and forms an estimate $\\{\\hat{\\theta}_k\\}$ of the unknown actual success probabilities $\\{\\theta_k\\}$ of `Bandit` environment. It takes the action with  $\\hat{\\theta}^* = \\max_k\\{\\hat{\\theta}_k\\}$ with $1-\\epsilon$ probability(exploit) and takes a random action with probability $\\epsilon$ (explore)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EGreedyAgent(RandomAgent):\n",
    "    def __init__(self, n_actions=5, epsilon=0.01):\n",
    "        super().__init__(n_actions)\n",
    "        self.epsilon = epsilon\n",
    "    \n",
    "    def get_action(self):\n",
    "        estimates = self.success_cnt / (self.success_cnt+self.failure_cnt+1e-12)\n",
    "        if np.random.random() < self.epsilon:\n",
    "            return np.random.randint(0, self.n_actions)\n",
    "        else:\n",
    "            return np.argmax(estimates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UCB1 Sampling Agent\n",
    "The agent tries different actions and records the number of successes ($\\alpha_k$) and failures ($\\beta_k$). \n",
    "\n",
    "It calculates the estimates of success probabilities:\n",
    "$$exploit = \\hat{\\theta}_k = \\frac{\\alpha_k}{\\alpha_k + \\beta_k}$$ \n",
    "\n",
    "It also calculates the exploration which favors the less visited actions more:\n",
    "$$explore = \\sqrt{\\frac{2.\\log{t}}{\\alpha_k + \\beta_k}} $$\n",
    "\n",
    "The agent then picks the action with maximum of explore + exploit score: \n",
    "$$score = \\frac{\\alpha_k}{\\alpha_k + \\beta_k} + \\eta\\sqrt{\\frac{2.\\log{t}}{\\alpha_k + \\beta_k}} $$ \n",
    "where $\\eta$ controls the relative importance of explore vs exploit. We will use $\\eta=1$ in our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class UCBAgent(RandomAgent):\n",
    "    def get_action(self):\n",
    "        exploit = self.success_cnt / (self.success_cnt+self.failure_cnt+1e-12)\n",
    "        explore = np.sqrt(2*np.log(np.maximum(self.total_pulls,1))/(self.success_cnt+self.failure_cnt+1e-12))\n",
    "        estimates =  exploit + explore\n",
    "        return np.argmax(estimates)        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thompson Sampling Agent\n",
    "\n",
    "This is based on Bayesian Approach. We start with an agent which has uniform probability (prior) which is reflected with a prior beta distribution of $Beta(\\alpha=1, \\beta=1)$. The agent then samples the actions using this distribution. And then it picks the action with maximum $\\hat{\\theta}_k$\n",
    "\n",
    "As the success and failure counts are updated in each step, the agent updates it belief of the $\\hat{\\theta}_k$:\n",
    "\n",
    "$\\hat{\\theta}_k \\sim Beta(\\alpha=\\alpha_k+1, \\beta=\\beta_k+1)$ \n",
    "\n",
    "It uses the updated $Beta$ distribution to sample and pick the action with maximum $\\hat{\\theta}_k$. And the cycle goes on\n",
    "\n",
    "It is basically picking up actions based on the samples of the belief of which action has the what chance of being best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ThompsonAgent(RandomAgent):\n",
    "    def get_action(self):\n",
    "        estimates = np.random.beta(self.success_cnt+1, self.failure_cnt+1)\n",
    "        return np.argmax(estimates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Playout\n",
    "`get_regret` calculates the total regret for a given agent and an environment with fixed $\\theta_k$'s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_regret(env, agent, n_steps=10000, n_trials=10):\n",
    "    score = np.zeros(n_steps)\n",
    "    optimal_r = env.optimal_reward()\n",
    "\n",
    "    for trial in range(n_trials):\n",
    "        agent.reset()\n",
    "        for t in range(n_steps):\n",
    "            action = agent.get_action()\n",
    "            reward = env.pull(action)\n",
    "            agent.update(action, reward)\n",
    "            score[t] += optimal_r - reward\n",
    "    score = score / n_trials\n",
    "    score = np.cumsum(score)\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot\n",
    "We plot the total regret curve for all the three exploration strategies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_regret(agents, scores):\n",
    "    for k,v in agents.items():\n",
    "        plt.plot(scores[k])\n",
    "    plt.legend([k for k,v in agents.items()])\n",
    "    plt.ylabel(\"regret\")\n",
    "    plt.xlabel(\"steps\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Everything put together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEGCAYAAABhMDI9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABGnElEQVR4nO3dd3gU1frA8e9JSEjohN5D76H3SJcuoCJIERBsXLxcsAGKgqiI5aeIYOMCF5SmdFF6EYP03gklQKiBkEAICcnu+f0xm80uaZuQzaa8n+fh2ZkzZ2bfCcm+O2fOnKO01gghhBBx3FwdgBBCiMxFEoMQQgg7khiEEELYkcQghBDCjiQGIYQQdnK5OoDHUbRoUe3r6+vqMIQQIkvZv3//La11saS2Z+nE4Ovry759+1wdhhBCZClKqYvJbZemJCGEEHYkMQghhLAjiUEIIYSdLH2PITExMTEEBwcTFRXl6lBEDufl5UXZsmXx8PBwdShCpEq2SwzBwcHkz58fX19flFKuDkfkUFprbt++TXBwMBUrVnR1OEKkSrZrSoqKiqJIkSKSFIRLKaUoUqSIXLmKLCnbJQZAkoLIFOT3UGRV2a4pSQghspWjS+HYcshXHIrXgiYvgZtzv9NnyysGV3N3d6d+/frWf1OnTk31Mfbt28eoUaMA+N///sfrr7+e5ngCAwPp0aMHlStXplGjRrRr147t27en+XjJadu2bYoPHT548IA2bdpgMpmcEkNS5s2bR9WqValatSrz5s1LtE50dDT9+vWjSpUqNGvWjKCgIOu2Ll26UKhQIXr06GG3z/PPP09gYKAzQxc51cP7sGw4nP4D9s+FtW/D5MJGuRPn0pErBifw9vbm0KFDj3WMxo0b07hx48eOJSoqiu7du/Pll1/Ss2dPAI4dO8a+ffto3bq1Xd3Y2Fhy5XL+r8ScOXN45plncHd3d/p7xQkNDeXDDz9k3759KKVo1KgRPXv2pHDhwnb1Zs+eTeHChTl79iyLFy9m7NixLFmyBIC3336byMhIfvzxR7t9RowYweeff86sWbMy7HxENhMbDUd+herd4NxmWP4y5C4I0eGJ159SGlr+Gzp97JRw5IohA/n6+jJ27FiaNm1K06ZNOXv2LAC//fYbderUoV69etYP623btiX4Zgpw8eJFOnTogJ+fHx06dODSpUsADB06lFGjRtGyZUsqVarE0qVLAViwYAEtWrSwJgWAOnXqMHToUAAmTZrEK6+8QqdOnRg8eDAhISE8++yzNGnShCZNmrBjxw4A7t+/z7Bhw2jSpAkNGjRg1apVgPHt//nnn8fPz49+/frx4MEDwPiAHTNmjPU9Z82axRtvvGGNqVevXtZtX3zxBU2aNMHPz4+JEycCEBQURI0aNRgyZAh+fn706dOHyMhIAMaNG0etWrXw8/Pjrbfecuhnv379ep588kl8fHwoXLgwTz75JOvWrUtQb9WqVQwZMgSAPn36sHnzZuJmOezQoQP58+dPsM8TTzzBpk2biI2NdSgWkcOZzWCKsS/7uDisfh2+qGQkBbBPCu9ehbFBMCEkvuzoUqeFmK2vGD78/Tgnrt5N12PWKl2AiU/VTrbOgwcPqF+/vnV9/Pjx9OvXD4ACBQqwZ88e5s+fz+jRo1mzZg2TJ09m/fr1lClThrCwsGSP/frrrzN48GCGDBnCnDlzGDVqFCtXrgTg2rVrBAQEcOrUKXr27EmfPn04fvw4DRs2TPaY+/fvJyAgAG9vbwYMGMCYMWPw9/fn0qVLdO7cmZMnT/LJJ5/Qvn175syZQ1hYGE2bNqVjx478+OOP5MmThyNHjnDkyBHre8Uli88//xwPDw/mzp3Ljz/+yMOHDzl//jxxgx9u2LCBwMBA9uzZg9aanj17sn37dsqXL8/p06eZPXs2rVq1YtiwYXz33XcMGzaMFStWcOrUKZRS1p/XggUL+OKLLxKcW5UqVVi6dClXrlyhXLly1vKyZcty5cqVBPVt6+XKlYuCBQty+/ZtihYtmuTPz83NjSpVqnD48GEaNWqU7M9a5GD75sI/0yH0vLH+xinwLgSflExY1zMfPIwwlscHg2deIK+xPvooHFoEbd5xWqjZOjG4SnJNSf3797e+xn2jbtWqFUOHDqVv374888wzyR57586dLF++HIAXXniBd96J/+Xo3bs3bm5u1KpVixs3biS6/9NPP01gYCDVqlWzHqdnz554e3sDsGnTJk6cOGGtf/fuXe7du8eGDRtYvXo1X375JWA0UV26dInt27db74X4+fnh5+cHQN68eWnfvj1r1qyhZs2axMTEULduXa5evUqhQoWsx9+wYQMbNmygQYMGAERERBAYGEj58uUpV64crVq1AmDQoEFMnz6d0aNH4+XlxUsvvUT37t2tV1UDBw5k4MCBSf7cEpvbPLFeQ47We1Tx4sW5evWqJAaROLMJ1oy2L/uqhv36c/PAHAu1n0n+5nKh8tB2bLqHaCtbJ4aUvtm7gu2HTNzyDz/8wO7du/njjz+oX79+qu5P2B4vd+7c1uW4D7jatWvb3WhesWIF+/bts2uCyZs3r3XZbDazc+dOa6KwPd6yZcuoXr16sjHYeumll5gyZQo1atTgxRdfBIykadu3X2vN+PHjefXVV+32DQoKSnBcpRS5cuViz549bN68mcWLFzNjxgy2bNmS4hVD2bJl2bZtm7U8ODiYtm3bJqhftmxZLl++TNmyZYmNjSU8PBwfH59Ez89WVFRUgp+ZEAB8WR0irtuXlWsGl3fHr3ecBLV7Z2RUyZJ7DBks7kbmkiVLaNGiBQDnzp2jWbNmTJ48maJFi3L58uUk92/ZsiWLFy8GjOYTf3//ZN9vwIAB7Nixg9WrV1vL4trqE9OpUydmzJhhXY9LUp07d+bbb7+1JpyDBw8C0Lp1axYsWAAYN7WPHDli3bdZs2ZcvnyZhQsXWq+UChcujMlksiaHzp07M2fOHCIijMvmK1eucPPmTQAuXbrEzp07AVi0aBH+/v5EREQQHh5Ot27dmDZtmjW+gQMHcujQoQT/4u61dO7cmQ0bNnDnzh3u3LnDhg0b6Ny5c4Lz79mzp7XH0tKlS2nfvr1DVwxnzpyhdu3M90VEuFDUXZhU0D4pvH8bJoXD8A0wMQzKNoH2E8B/TJKHcYVsfcXgKo/eY+jSpYu1y2p0dDTNmjXDbDazaNEiwOjtEhgYiNaaDh06UK9ePf76669Ejz19+nSGDRvGF198QbFixZg7d26ysXh7e7NmzRreeOMNRo8eTYkSJcifPz8TJkxI8vgjR47Ez8+P2NhYWrduzQ8//MD777/P6NGj8fPzQ2uNr68va9asYcSIEbz44ov4+flRv359mjZtane8vn37cujQIbveP506dSIgIICOHTvSqVMnTp48aU2S+fLl45dffsHd3Z2aNWsyb948Xn31VapWrcqIESMIDw+nV69eREVFobXm66+/Tv4/w8LHx4f333+fJk2aAPDBBx9YrwQ++OADGjduTM+ePRk+fDgvvPACVapUwcfHx5qEwbjJfOrUKSIiIihbtiyzZ8+mc+fO3LhxA29vb0qVKuVQLCKb0ho+LJT4tho9oOUocLf5yFUKXtqUIaGllkqsTTWraNy4sX60z/zJkyepWbOmiyJKXtzEQsndyMxuevTowZgxY+jQoYO17ODBg3z11Vf8/PPPSe4XFBREjx49OHbsWEaE+Vi+/vprChQowPDhwxNsy8y/j+IxRIVD4Eao86zxAQ8wvSGEnktYd2JYfJ1MQim1X2udZH94uWIQThHXc6levXp2SQGgQYMGtGvXDpPJlKHPMjhLoUKFeOGFF1wdhsgIO7+D9ePj15dZvgy4eYD5kS6o7SZAm7czLrZ0JFcMQjiR/D5mcWYzxERC7nwQHQGflkm+fqdPoGXaRynIKHLFIIQQjgg5DceWwd7/QuTtpOuVaQQvb4HIUDi1BkwP4Y83oddMaDAo4+J1IkkMQggRegFmNk25HsBLm43XPD7QcLCx3OQl58TlItJdVQiRc0Xfg5unYHp9+/IyllaWkn7wyrb48jdOZbobyc4gVwxCiJwlMhS8CsKNY/Cj/UCSjL8Ct04bzUW2JiUxmF02JVcMThAUFESdOnXsyiZNmmQdTuLLL7+kRo0a1oHz5s+fDxhDVlevXp369etTs2ZNfvrppwyPXYhsS2uYUgY+rwiTfRImhdcCjJvMjyaFHMhpVwxKqXLAfKAkYAZ+0lp/o5TyAZYAvkAQ0Fdrfceyz3hgOGACRmmt1zsrPlf54Ycf2LhxI3v27KFAgQKEh4dbB8ED42nmxo0bExoaSuXKlRk6dCienp6uC1iI7CL0fPzAdHFyecN713JE81BqOLMpKRZ4U2t9QCmVH9ivlNoIDAU2a62nKqXGAeOAsUqpWsDzQG2gNLBJKVVNa52xs7k42ZQpU9i6dSsFChQAoGDBgtZhnm1FRESQN2/ebNHPXwin0tp44My7kH252QQXd8Ci/vYJ4Zn/QuEK4O4BpRtkaKhZhdMSg9b6GnDNsnxPKXUSKAP0Atpaqs0DtgFjLeWLtdbRwAWl1FmgKbAzzUGsHQfXj6Z590SVrAtdUz8jGxhDZdy7d4/KlSsnWWfgwIHkzp2bwMBApk2bJolBiOScXgeLjCHteX0fFCxrdDvd/oXRlTQxdfvIFUIKMuTms1LKF2gA7AZKWJIGWutrSqnilmplgF02uwVbyh491ivAKwDly5d3YtRpl9Sga2azOcUB2eKakkJCQmjZsiVdunShQoUKzghTiKxt1/ewblz8+oxkZjzMXQCi70LfnyUpOMDpiUEplQ9YBozWWt9N5oMxsQ0JHsvWWv8E/ATGk8/Jvnkav9k/riJFinDnzh27stDQUBo1akTevHk5f/48lSpVSvYYxYoVo2HDhuzevVsSgxCJiUsKXgWNpqRH+VSCJ96CBknP0yES59ReSUopD4yksEBrvdxSfEMpVcqyvRRw01IeDJSz2b0scNWZ8TlLvnz5KFWqFJs3Gw/ChIaGsm7dOvz9/Rk/fjwjR47k7l1jZrm7d+8m2vsoMjKSgwcPJtvsJESOozU8CDOGswZjjKJxl2DCTXh5qzG72ZgTRvfSUQclKaSRM3slKWA2cFJr/ZXNptXAEGCq5XWVTflCpdRXGDefqwJ7nBWfs82fP5+RI0fy5ptvAjBx4kQqV67MiBEjiIiIoEmTJnh4eODh4WGtA8Y9Bm9vb6Kjoxk6dKjMCCYEQPB+2P45nHlknu4+s43XXLmhTPJT2ArHOW0QPaWUP/A3cBSjuyrAuxj3GX4FygOXgOe01qGWfd4DhmH0aBqttV6b3HvIIHois5Pfx8ekNXzjB2GXEm7r+CG0+o/cM0gDlw2ip7UOIPH7BgAdEivUWn8CfOKsmIQQWcDRpfHDWSem0VBo+gqUkBnznEWGxBBCZA6mGPgoiUms2oyDduMT3ybSnSQGIYTrfVrO6E6amKqdJClkMEkMQgjX2ToF/vrMvqzHNGNeA3cP495C/tIuCS0nk8QghEh/WsPnleBBqLFerCbkLwmDVxrrETdhQR+4djh+n85ToPm/7G8mF8qcD7Fmd5IYhBDp77sW8UkBIOSk8S/u+YNHvRZgDDcjMgVJDOns9u3bdOhgdLq6fv067u7uFCtWjKCgIEqXLs2JEydcHKEQj+n4CvhtKIy7DF7GYJCYYmDjB7DrO/u6T30Dv/8n6WMNWw/lmzstVJE2khjSWZEiRTh06BBgzMGQL18+3nrrLYKCgujRo4drgxPice2bA2vGGMtTy8VPYJNYb6JuXxpdSxsNNdaD98N/2xvLQ9ZAxSecHa1II5moJwOZTCZefvllateuTadOnXjw4AEAhw4donnz5vj5+fH0009bx1lq27YtY8aMoXXr1tSsWZO9e/fyzDPPULVqVSZMmAAYkwLVqFGDIUOG4OfnR58+fYiMjARg3Lhx1KpVCz8/P9566y0ALl68SIcOHfDz86NDhw5cumQ8ODR06FBGjRpFy5YtqVSpEkuXLs3oH4/I7LZ8HJ8U4kwqmLB5qNPHxrAUTV+2Ly/byEgkk8IlKWRy2fqK4bM9n3Eq9FS6HrOGTw3GNh2bpn0DAwNZtGgRs2bNom/fvixbtoxBgwYxePBgvv32W9q0acMHH3zAhx9+yLRp0wDw9PRk+/btfPPNN/Tq1Yv9+/fj4+ND5cqVGTPG+CM9ffo0s2fPplWrVgwbNozvvvuOYcOGsWLFCk6dOoVSirCwMABef/11Bg8ezJAhQ5gzZw6jRo2yThR07do1AgICOHXqFD179qRPnz6P++MS2UHUXVg5In4Y6zp9oM1YmNnEvl6v72RsomxCrhgyUMWKFalfvz4AjRo1IigoiPDwcMLCwmjTpg0AQ4YMYfv27dZ9evbsCUDdunWpXbs2pUqVInfu3FSqVInLly8DUK5cOVq1agXAoEGDCAgIoECBAnh5efHSSy+xfPly8uTJA8DOnTsZMGAAAC+88AIBAQHW9+rduzdubm7UqlWLGzduOPeHIbKG2IdGk5Ht3AZ9ZkOxavDvA8Z6ibrwr12SFLKRbH3FkNZv9s6SO3du67K7u7u1KcmRfdzc3Oz2d3NzIzY2Fkg4/4NSily5crFnzx42b97M4sWLmTFjBlu2bElwfNt9bY/vrDG0RBbzcTH79Ylh8ctFKsffYxDZilwxuFjBggUpXLgwf//9NwA///yz9erBUZcuXWLnTmOiu0WLFuHv709ERATh4eF069aNadOmWW+It2zZksWLFwPGpED+/v7pdzIie/ntxfjlwauMJCAD1uUI2fqKIauYN28er732GpGRkVSqVIm5c+emav+aNWsyb948Xn31VapWrcqIESMIDw+nV69eREVFobXm66+/BmD69OkMGzaML774gmLFiqX6vUQOcessHLdModL3Z6jU1qXhiIzltGG3M4IMu421G+yxY8dcHYpIRJb9fVz+KhxZDPUGwNPfuzoakc5cNuy2ECILun8bDv0CgRugTGNJCjmUJIYsztfXV64WRPq4uBPmdolfL9TWZaEI18qWN5+zcvOYyD6y3O+hbVIAY0A7kSNlu8Tg5eXF7du3s94fpchWtNbcvn0bLy8vV4fimJDT8csDfoMGL0C5JknXFy7z4KGJiOhYp75HtmtKKlu2LMHBwYSEhLg6FJHDeXl5UbZsWVeH4Zg9Pxmvr+2AknWgWifXxiPsaK2ZuPo4u87f5syNCADaVS/G3BebOuX9sl1i8PDwoGLFiq4OQ4isI+Q07P2vsSzzKGc6Wmsqjv8zQXmv+mWc9p7ZLjEIIRyw4jU4vMi+rEIreYAtE3ry6+1263XKFGDNv507CKEkBiFyiuh7cP8WFPZNmBQAXkz4rVS41v6LoZy9aTQdnZjcmTyeGfORLYlBiJwgMhRmtYM7QYlvfy0g8XKR4aJiTADUeH+dtWxslxoZlhRAEoMQ2ZvWcOsMzEzkJuWbZyB/iYyPSSQq1mQmxqSp+cE6u/IyhbwZ0bZyhsYiiUGI7OT4SvhtSNLbO38Kkbeg5SjwLpRRUYkUaK2p8t7aBOWb3mhDleL5MjweSQxCZEV3gqBQBfubxY/OpPao92+De879kw+LfEh+Lw/c3Zx7g31vUCgms6Z5pSJJ1vlqw2mmbzkLwFudqvHlhjN225tW9GHK03VckhRAEoMQmc+VA8b9AEe9dwM+eaRJqNVoOLwYukyBOs+ma3hZzcNYM9UmGN/G3+5cnZHtqgDGg2IxZjMFvDyS3Ndk1lR+90+GtKjAh73qJNg+c+tZvlhvPBx4ZFIn/CZtsNs+vX8DetYrbV1PrOupbVJYP7o15Xy8M/R+QmKy3eiqQmRZYZfh4C/w19S0H6P391B/QPrFlMVFxZjsbuICeLgrYkz2n3tNK/qw8KVm5HI3BoOINZnRQFWb5p2+jcvyeZ96aK3ZdiaECSuOcSUs5cm2vhvYkH8tOJBsnYLeHswf1pR65Qo5dmKPKaXRVSUxCOFqt8/Bgucg9Fzi21u8DmYTVO8Kl3fD1k9gwK9w9RBsmxJf7/mFUKN7hoScFST1YFhKPu5dhwkrEx+Yclq/+oxecijZ/f9+px2lC3nTedp2a1fTR639zxPULFUg1bGlF0kMQmQGUXfh91FwfAWMuwxelg+FOV3g0s6E9SfcBDcPiL6b/E3iezcg4joUr52j7x/Y0lqz/vgNpm06w6nr9wA4N6Ubt+9H0/STzXZ1f3/dn6dmJN9V9z8dqnLh1n1WH76a6PZzU7rhpuBuVCwFveObpSKiY6kzcb11vWQBL67fjeK311rQxNcnraeXLiQxCOEqZ9bDwr6Jb6v9TPwMabbGB0Pu/M6NK4vTWhN85wFbTt1k4urjbH+7HZtP3aBQHg/GLDmcoP7f77SjnE8eAG5FROOmFD55PdFaW+c8vx4eRfNPN5PH053Ih8ZzBB/0qEWzSj7ULl2Qe1ExtJq6hbtRxuB1nz1bl35NymfQGac/SQxCpLcTq8G7MFS0DEugtXElUKkt5LF8E7yyH2a1d+x49QdB75lOCTWrMZs1G05cp2314nh5uBNjMjNg1i4OB4fzMNbMdwMb8uWG05wPue/Q8b58rh59GqV+IEOzWeOWSO+lgMBb1C1b0O7KICuSxCBEeokIMZp1PioaXzZyDxyYDztnGOvPzoYSdeC7ZsZ64YoQfhlq9TK2KWXfrbRSWxi8KqPOINPzHffHYx9jySvNaZZMV1EhiUGIxxMdYdwbyFMU9vyY+v0nhad/TFnUvagYPHO5EXr/IS0+3cL4rjV4tY3xRG9abhQXz5+bOUOb0OPbAJr4FiaXmxufPlMX36J5nRF+tiKJQYjHkdRDY0N+h3lPxa93/8roanrVpltisxHQ9TG6nmZBWmu2B95i6f5gfj98Ff8qRfm2fwN2nr+dYpfNOJ8+U5di+XIzY+tZroQ9YOOY1hTK48mVsAe0mroFgJOTu+Dt6e7MU8nWJDEIkVYPwuCzCvHr+UuBKQYG/gplGhn3FkwxcD8ECpYBU6zRw6iUH3jmB7dsN0FiijaduMFL89P+N5lSj53Dl8MIfxBD62rF0vweIuXE4LT+bUqpOUAP4KbWuo6lbBLwMhA3vdq7Wus/LdvGA8MBEzBKa70+wUGFyEh7Zhmvz86Gun0SblcKcnkaSQGM7qIVnTtOfmYWEHjLoaTg7eHO0UmdcHdT3ImMYcXBK3y05gTPNCiTYjfOjHoALKdz2hWDUqo1EAHMfyQxRGitv3ykbi1gEdAUKA1sAqpprU3JvYdcMQinMZtgsuVDasJNyJXbtfFkYjvO3mLnudvM2HrWWhY01XjQLujWfYrlz01UjIki+eRnmFm47IpBa71dKeXrYPVewGKtdTRwQSl1FiNJJPLkjxDp4OF98MhjPwhd7EP42NJEoWyagSQpJNBzRgBHghO/sb5zfHw33bgbwXlzy8N3WYkr/rdeV0oNBvYBb2qt7wBlgF02dYItZQkopV4BXgEoXz7rPmAiMtCvQ+DEyoTlFVsbN5HjTKsbv6zNxuu4S04NLSsKvHEvyaRw9pOu1vGGRNaV0f+D3wOVgfrANeD/LOWJjYObaBuX1vonrXVjrXXjYsXkBpRIgSkm8aQAcGE7bPrQWI66awwtYeuJt8ArhaGscyDbOYh71y/N+SndCJranaCp3SUpZBMZesWgtb4Rt6yUmgWssawGA+VsqpYFEh+YRIjkBG6EBZYbxe9egymlEq/X4AU4+DMEfGX8i9N4GPT42vlxZlGrDl2xLsfdRxDZT4YmBqVUKa31Ncvq00DcEIargYVKqa8wbj5XBfZkZGwiGwiYBpsmxq/bJoWJYfb3EwBio+Hor/ZlkhQSdSsimrDIGP6z+BAAP77QyLUBCadyZnfVRUBboKhSKhiYCLRVStXHaCYKAl4F0FofV0r9CpwAYoGRKfVIEiIB26Rgq/2EhEkB4NlZ0OljmN/T6IU0Ur6LPCr0/kNWHrzC5DUn7Mo71y7poohERpAH3ETWd/82fFHJWC5UAUYfgYX94Mw6KFIV/i2/I6kRN+NZ80o+7DofmmD7kUmdkp31TGR+LuuuKoTTrHvXGHqi62dQtFp8UgDov9h4HbDENbFlA8//ZPQSfzQpSI+jnEMSg8habhyHXZYhqn9sbb/t3WvgmSfjY8pGjgSHceBSmF3Z/gkd5eG0HEYSg8h8fvCH60fty17fZ4xN9H3LxPcZd1mSQhrdi4ph4e5LDPevSM8ZOwD4qFdtmlUqwr2oWEkKOZAkBpG57P9fwqQAMOOR5tB3LsDSYXB+K3SeEj9VpkiVPRdC6fuj0XT06dpT1vJBzStYZzcTOY8kBpE5aA0fFnKs7gd3jJFLB690ZkTZgsmsiTGZ2XnuNi/+by8A/3uxCUPn7k1yn2MfdpakkMNJYhCZg+3TyU9Ohlb/sd8eGWpcSfj658jhrFMrLPIhx67cZdDs3Qm2JZYUAj/pSozJTO5c7rgnMqWlyFkkMQjXu7wXfhtqLL+4Fiokch8hjw9UapOhYWVVry88wJoj11KuCPRvWp4pT9dBKYWH9DgSFpIYhGvdOA6zO8avJ5YUhMP6/riTPRfsu5n+MrwZ/lWLJrGHEAlJYhAZ79gy48bxqIPxvYwKljfWRZqZzNqaFIa0qMD4bjXx8pDpL0XqSWIQGW/pMON1eoP4sn/9Y8yAJlJFa01EdCyh9x/S5ottAIxoW5mxXWq4NjCRpclfonA+rWFuN7j0T+Lbx1+B3PkyNqYsLizyIfUnb0x02386VM3gaER2I4lBOFf4Ffi6VsLywauNZFC6YeID3IlEmc2aSu/+meT2P0c9Ic1H4rFJYhDOExFinxTq9oU7QdBgkPQwSqN5O4Ps1r/t34DudUuhFPLsgUg3khiEc0Tfgy+rxK//azcUl3bvxxFjMvPh78bw1w3KF+KZBmV4ql5pF0clsiNJDCL9mU3wadn49UHLcmRS0Fqz4uAVutQpiYe7G6ev36NmqQKYzJpqE9bi5eHG3vc6kj+FIawfPDRR84N11nV3N8WKf7VydvgiB5P5GET6Mptgsk/8+nvXwcPbdfG4gNaaupM2EBEd61D9s590Zf7Oi0xec4K973Xky/WnqVO2IB5uikYVCtvNsRxXX4a/Fo8jXeZjUErl1lpHp1QmBD/Z3Dt4dnaOSQq3I6J55ef9uLupBA+YpaTKe2uty00+2QTAkn2XE9Sb8nRd6pcrJElBOJ2jTUk7gYYOlImcLHBj/Mio71wwhrHIZh48NNFsyibuRjl2NbBu9BPM33mRhbsvUa9cIfo1Lsfxq+F0r1uK5pWKJNvDyNbKka2oX67QY0QuhOOSTQxKqZJAGcBbKdUAiOv2UACQwe+FvT/fMl6LVM1WSSGlLqKPKpovN2v+7U/Jgl6A8U3/gx618HR3w+2RAeq2vdWWtl9uY82//alTpiBnbtyjbGFvtpy6yZO1ShAh8yEIF0j2HoNSaggwFGgM2Dbm3wXmaa2XOzW6FMg9hkzAFGM8yXxydXzZpHDXxZPO7kfHUnvi+mTrTHm6Lt3rliJvbndp5hFZwmPdY9BazwPmKaWe1VovS/foRNZlNkP4ZfjGz7689w+uiSeV1hy5yp37D3l/1XFrWZlC3rSpXoyoGBPju9a0tvc/6sTkzuTxlA59Ivty9Ld7h1JqNlBaa91VKVULaKG1nu3E2ERmYjaBm+WJ2pO/w5JBCev0mgn1+2dsXGkQHWvi9YUJB+y7EvaAhbsvAbD8wBW7be91q8nLrStlSHxCuJqjiWGu5d97lvUzwBJAEkN2F3YJptU1lks3NCbQ+W2IfZ23z0PeIhkfWxporWk2ZbNdWZtqxahcLB9zdlywK8/vlYsefqX4oEdtvD1lmAmRczj0HINSaq/WuolS6qDWuoGl7JDWur6zA0yO3GNwshsn4PsWSW8fuQeKVc+4eB7TW78dZun+YOv6mY+74pkr4T2BwBv3CLkXTcsqMoeByJ7S5TkG4L5SqgigLQdtDmSfO4wi3t2r8FXNhOUDl8KCPvHrWegGs9aaiuPtexX9MrxZokkBoGqJ/FQtkT8jQhMiU3I0MbwBrAYqK6V2AMWAPsnvIrKUgGmwaWLi2+KSwIQQ2PMTNH0lw8JKDdsEsGt8B0oW9OJhrJlqE9ba1fvyuXoyo5kQyUgxMSil3IE2ln/VMZ5lOK21jnFybMKZYqIg8jYULANHfk2YFJ5faDQl+Y+OL8vlCS1fz9AwHRH+IIb8uXPZPWvQ/NPNCer1rFea6f0bJCgXQthLMTForU1KqV5a66+B4ynVF5lc8D74b4ektw9ZA+VbGLOp1eiecXGl0e7zt+n3064U6x2Z1IkCKQxWJ4QwpKa76gyMnkj34wq11gecEpVwDlNs0kmhXDMYviFj43kMvuP+SLR8Qvea9G9anld/3k/HmsXpWb8MD2PNkhSESAVHE4NlxnYm25RpoH36hiOc6qNHupQO+BVWjYSun0HtZ1wTUyporXl/1TF+2XXJrtzT3Y0//+NPpaL5rENO/PJSM1eEKES24FBi0Fq3c3Ygwolio+Hj4vHr4y6DVwFj+e2zronJAXejYth+JgRPdzde+Xl/gu1PNyjDS09UpHbpgi6ITojsy9Fht99IpDgc2K+1PpSuEYn0t3hA/PKY4/FJIRNoPmUz1+9GETS1O2azJuj2ffYGhdK5dskkJ7sH+PuddpTzkXEchXAGR5uSGlv+/W5Z7w7sBV5TSv2mtf7cGcGJx/TbUDi+In6910woWDbJ6hnp0WcLHr1nMHbZ0QT7vPlkNf7doarTYxMip3M0MRQBGmqtIwCUUhOBpUBrYD8giSGz+aYe3AmKX+/2JTRIZHyjDBQVY2LTyRss3H2Jf87ddmifXvVLU6loPv7TURKCEBnF0cRQHnhosx4DVNBaP1BKySxumc3dq/ZJoVgNaPqyy8IBY06DKX+eZP7Oi3bly0a0IDrWzIwtZ2lQvhBVi+encF5PhszZw8H3n6RwXk8XRSxEzuVoYlgI7FJKrbKsPwUsUkrlBU4ktoNSag7QA7ipta5jKfPB6PLqCwQBfbXWdyzbxgPDARMwSmud/CD4ImlxQ1p45od3g5Ov6wRaa1pN3cLV8Khk69mOVdSysv2TyEFTM/8zFEJkVw4NogeglGoE+GM8+RygtU529DqlVGsgAphvkxg+B0K11lOVUuOAwlrrsZZhvBcBTYHSwCagmtbalNx7yCB6j3h4H6aUjl/PwPGMjl0Jp8e3ASnW+3l4U56oWiwDIhJCJCW9BtED8Abuaq3nKqWKKaUqaq0vJFVZa71dKeX7SHEvoK1leR6wDRhrKV+stY4GLiilzmIkiZ2piC9nu38bvrCZL6D/khR32RcUyoSVx+jftDwTV9s/1G47x3DgjXs8+fV2fh7elKYVfcidy51Yk9k6W9nIhQf448i1BMf/9Jm6LN5zicPB4bzWpjL/aldZHjQTIgtwtLvqRIxeSdUx5mXwAH4BWqXy/Upora8BaK2vKaXiOteXAWzHNQi2lAlH2SaFD0LjJ9VJRIzJTNX34geWezQpAPSeuYO3O1cnIjqW77edA+CF2Xvs6rzbrQaNfX3skkLfxmUpmi83b3eujlKK/k3Lp/WMhBAu4ugVw9NAA+AAgNb6qlIqPcclVomUJdrGpZR6BXgFoHx5+dAB4LLNB/a7VxMkhX/O3mLAf3czb1hTVh+6yrIDid93+LpfPf63I4jDwUYT1BfrTyf7tlP+PGVdHt+1Bq+2qZzGExBCZCaOJoaHWmutlIqbjyFvGt/vhlKqlOVqoRRw01IeDJSzqVcWuJrYAbTWPwE/gXGPIY1xZC+znzReX1wHnvH/NY+2+w+ZY/+Nf9MbbahSPB8bjl+naUUfCuXx5OkGxnMOry88wBrLlcC/2lbmtbaVOXgpjBUHgll56CpPVC3K34G3AChb2FuSghDZiCPDbitgjVLqR6CQUuplYBgwKw3vtxoYAky1vK6yKV+olPoK4+ZzVWBPokcQxJrMDJi1mz1BoYyrcIbXLOX3SzQmL8acxi/O3ZvsswLvdKlOleL5AOhUu2SC7TMGNGRc10jOhdynTTXjZnGbasVoU60Y0543hq6eudUYTmNkuyrpd3JCCJdzZNhtrZTqjXGT+C7GfYYPtNZJj1cAKKUWYdxoLqqUCgYmYiSEX5VSw4FLwHOW9ziulPoVo+trLDAypR5JOZXWmiqW+wP11VleuzEJgK2mevzfrF2Mal81wbhC/ZuW59Nn6hIRHcu+oFDaVi/+6GETVbZwHsoWTnrYCUkIQmRPjs75PBP4n9Z6r/NDclxO6q667fRNhs61//EHecWPgeQbtTDR/eR5ACHEo9Kru2o74FWl1EXs52Pwe8z4hANGLjjAH0fje/54E8XG5wvBSmM96u3LfHg41K530YH3n8RHnhoWQqSBo4mhq1OjEEn68PfjdkmhW6HLfBc11poUGHUQr7wFGNKyAH0bl+OPo9foXrcU3p5Jd1cVQojkODofw8WUa4n0VvndPzGZjaa+fo1K81lLM8yyGUK7QBnwiX9+wdvTnT6NMsfoqUKIrCs1Tz6LDNRq6hZrUpj6dB2eX+uXcMbtF9cm3FEIIR6TJIZMaMLKo1wJewDAR73r8PyBAfYVBi2DKh1dEJkQIieQxJDJNPxoI6H3jRHOx7QswgsFDsGNY8bGCTchV27XBSeEyBEkMWQivWbusCaFRXX20uLA15ZBSICmr0pSEEJkCDdXByAMe4NCOXw5DIDPWkGLs1/bV+j6WcYHJYTIkeSKwcVC7kXz5m+H2X4mBIWZoZ5b6adjjI35S8Gbp5I/gBBCpDNJDC4QazJbh7WwdcHLMidzXPPRmEQnxxNCCKeSpiQXGD4v4TAezYonMjSUm/z3CCEynlwxONnDWDNhDx7i7eFOfi8PTl+/x19nQgBjUpsi+XLzdqfquC3sYwxR2G8BLBkIPb5O/sBCCOEkkhic6Me/zvHp2vh7BO2qF2PraSMpdKldks/71DM2RIXD2U3Gco3uGTpXsxBCPEoSgxME3bpP2y+3JSiPSwoAP7zQyFjQGqZaZqLr8hmoxCazE0KIjCOJIZ3ci4qh6zd/E3zngV150XyeTH++AbfuP2TUooPk98rFoQ86GRvvXIRvbAaobf4aQgjhapIY0kF4ZAz1Jm9IUD5naGOaVSxC3tzGj7ld9WLk9/KIrxDwVfzyawEIIURmIIkhHfzvnyDr8sh2lRnS0pfi+b0S1LNLCgBHfjVeh2+CknWdGKEQQjhOEsNj0lqzYLcxKvmSV5rTrFIRx3bc9T3ERBrL5Zo4KTohhEg96Sj/mL7eFMjNe9HUK1fI8aQQGw3rxhnLPWc4LzghhEgDSQyPIcZkZvrmQADmv9jU8R2/a2G8FqsBDV9wQmRCCJF2khgew9hlRwAY1b4KBfN4pFDbIvoehJ4zlv+1y0mRCSFE2klieAzLD1wBYLh/pRRq2vjUMvVmp0/kmQUhRKYkiSGNBv13NwBPVC3q+NVC4Mb45SbDnRCVEEI8PkkMaWA2awLO3gLg02cc7GZ64wQs6GMsD14FHt5Oik4IIR6PJIY0+HrTGcB4ZqFs4TyO7fS95YYzCiq1dUpcQgiRHiQxpMG3W84CMLpjNcd2uH4sfvmD206ISAgh0o8khlTqNXOHddnD3cEf3w+tjNcha8DN3QlRCSFE+pHEkAorD16xzsu85t/+ju10Jyh+2dfBfYQQwoUkMTjofnQso5ccAuCb5+tTp0xBx3bc/z/jddRB6Z4qhMgSJDE4qPbE9dblXvXLOL5jUAB4+4BPKp51EEI8ll3XdjFt/zTM2uzqULIkSQwOuB8da12+8Gk3x3eMjoDgvVAjFfsIIRL4aOdH1J1Xl8i4gSdtRMZEUndeXerOq8uCkwu4/eA2L294mdnHZlNvfj1u3L/hgoizNkkMDuhtueG88OVmKEebgyJD4VPLlUW5Zk6KTIjs6e7DuzT5pQl159Wl/5r+/HrGGKJ+2+VtNF3QlIM3D1rrNlsY//c1dc9U2v7a1u5YHZd25NaDW06P+d7De9YElRHv50xKa+3qGNKscePGet++fU59D601Fcf/CcC5Kd1wd3MgMVzYDvOeil+fEAK5PJ0UoRBZm1mbeWrFU1y6dylV+33R+gt2XdvFssBliW6f2WEmIzePtK5XL1ydpT2XPlasAIF3Atl7fS8Dag6wli04uYCpe6ba1ZvWbhodynd47PdzBqXUfq114yS3S2JI3rmQCDr831808S3Mb6+1THkHsxkmF45ff/8WuDs4ZIYQOVDdeUmPHlAyb0mu379Ow+INOXDzQJL1ZnWaRbOSzYiIiaDlopZ81OojelfpzeW7l+m2Ir4pd/eA3eTxyMND00Mm/jORNefXWMtScv3+dfJ75qf5wuapOj8fLx/clTvrnl1Ho1+Mud6PDjmaqmOkt5QSg0zUk4LVh66iFEzv3yDlyg/C4LMK8etjgyQpiGxtReAKGpVoRPkC5dO0/4aghFPiftTqI/J75qd1mdZ42Pz9xJpj2XhxI118u+A3P36u9F+6/UK9YvUAyO+Z3+5Dt1yBchwdcpS5x+by1f6v7Jqd4sSVLe6xmNpFatttux9zH3flTlRsFE8ufTLZc6npU5PFPRbjptx4YvEThEWHARAaFQpgTQoAWy5tYcXZFQyuNZjGJRo73kSdQeSKIRkms6byu39SoUge/nq7XfKVL+6EuV3i1984BQVKOS02ITLKpH8msfXyVkY1GMXNyJvU8KnBgpML2H19t7XOkcFHUv3hNnz9cPZc3wNAwPMBFMztYBdwixhzDO7KHTeV8q1SrbVdMnlc37T7hm2Xt7Hi7AoANvbZSMm8Je3qPIh9wJ5re3h9y+spHm/XgF3WK5Hfe/+Ob0HfdIs1MZmyKUkpFQTcA0xArNa6sVLKB1gC+AJBQF+t9Z3kjuPsxHDsSjg9vg3g1daVGN+tZvKVJ9n8Un8QKk84iyzl0M1DzD8xH+9c3nzi/wlgfJj2+b0PZ+6cSdWxfu76M/WK1WPTpU08ND2ke6XuCerYtsl7uHlw4IWkm4nSy+5ru3lpw0uMqDeCp6s8Tal8xhe36/evp3g1EGdR90XU8KlBLjejsSXWHGtdTk60KZo5x+ZwNOQoh0IOce/hPYfeb2yTsQyqNQitdbpeVWTmxNBYa33LpuxzIFRrPVUpNQ4orLUem9xxnJ0Yxi49wpJ9l9n2Vlt8i+ZNumLoeZhuaWoa+oc84SyylLgPTFvD6gzj4t2LbL60Odl9W5dtzfbg7cnWmdlhJq3Ltrauf73/a+YcmwPAy3Vf5mW/l/HO5drRhu/H3Gf01tHULFKTaxHX8C/jz7QD02hWqhlFvYqyLmgdG/tsTJcPZ601u6/vpn6x+kSbovFf7Njnxapeqzhw8wC9q/QmLDqMot5F0xxDVkoMp4G2WutrSqlSwDatdfXkjuPsxOA77g/AeHYh2V+IbZ/BtinwyjYo7cC9CJEjHAk5wsA/BwKwuPtiahetncIeGScyJhKvXF5M3jk5yV49cfyK+rGg+wJMZhNuyi3B38KD2Accv3WcKxFXmLBjgsMxpFcvoazuZuRN8nnkwzuXNzcibzh89VI+f3n+eOaPNL1nZk0MF4A7gAZ+1Fr/pJQK01oXsqlzR2tdOJF9XwFeAShfvnyjixcvOiXGuG6qxfPnZs97HZOv/J2lt9K//nFKLCLruffwHi0XJd6LLSPakJNzPuw8vVb1SlB+ZPARDt48yJB1Q6xlXSt25fPWn6fq+IPXDsZkNvFLt1+SbNd3V+4cGnwoVcfNic6FnWPthbX8eOTHBNuG1RnGmEZj0nTczJoYSmutryqligMbgX8Dqx1JDLacecVwNewBLaduYXKv2gxu4Zt0xfBg+Lo21HwK+v3ilFiEawWFB+Gm3Mjllgt35U6JvCUS1DGZTQxdN5RDIYfoXaU3K8+udOjYfz79J+UKlANIl3bkGFMMywOXU7FgRZqWamo97kPzQzzcPPjn6j98tuczgu4GWffpVbkXH/t/bHecVWdXMWHHBHYN2EVej2SaUVNg1mbqza9nV1Y6b2mW9VxGPs98aT5uTnQ+7Dwebh7cjrpN7SK17XpspVam7K6qtb5qeb2plFoBNAVuKKVK2TQl3XRFbHF2XzDmTahXtlDSlQ4vhhWvGstVHLv8E1mL1pqnVj5lVzaqwSgG1hxIn9/70L5ce87cOcPOazut222TQlxvlZuRN1kWuIzvDn1ndyzbPva29VPLZDbRaVknbkam7s/mzUZvMrTO0ATlvar0oleVhFcVqeWm3FzeZz+7qFTIGG8t7ouEM2X4FYNSKi/gprW+Z1neCEwGOgC3bW4++2it30nuWM68YmjyySZC7kVzfko33JJ62tm2J5I83ZwpmLWZm5E3rR+utt0U3278NoNrD052/73X93I27CzdKnajYO6CTD8wnVlHZzn8/qXyluLa/WsA7Oi/gwKeBey2a62ZvGsyS88k3bZesWBFVvde7dD73bh/gxhzDF2Xd3U4RoDvO36PfxnpJJFTZbqmJKVUJWCFZTUXsFBr/YlSqgjwK1AeuAQ8p7UOTe5YzkwMcTeeg6Ym7GoHwJHfYLmlJ8ekcKfEIFInJDKE9r+1t64ffOEgAVcC+PeWfyeou7LXSnqv6g3A1r5bKeBZgLf+eoutl7cmeuzt/bYTbYpO9sZgYokgOTcjbzJy80im+E9hwckFCW4CD68znCG1h1DYy2hRPRxymEF/DrJuVyg09n+//ar347V6r7H2wlo+32vcG+ji24Xr969zPfI6S3osIb9nfjzc5MHLnCzTJYb05KzEcD4kgvb/9xcTn6rFi60qJl7pe3+4cRQGLoWq0ozkSqdCTzHz4Ey2BW9zyvFr+tTk16eMQdxizDHce3iP5YHLebLCk5TPXx6Ndughq5SYzCaGrR+WYOiHBd0WMGrLKG5HJT8t7OiGoxled/hjxyGyv0x5jyGzm/X3eQA6106irffMBiMpdPhAkkIGuBB+AZPZxLbgbbgrd77a/xX5PPKx7tl19FzZ0zrkQJzp7aYzausou7Ijg4849ORrpYKVWNZzGYdDDrPv+j6alGxCwxINrds93Dzw8fLhpbrx/f4V6fPgkbubO/O6zgPsxw+K6/KamF97/MpPR35iZP2RVClcJV3iEEKuGBLR+vOtmMyaHePaJ14h7t7C6KNQKG1jxIjkRcZEMvf4XH44/EOq9tvUZxMl8pZg17VdTNk9he87fk+ZfElPrGR7D2LtM2spm7/sY8Wdnh4dxmFO5zk0KdkEMJ64DYkMsT69K0RqyBVDKj14aOJSaCRDW/omXuHutfhlSQpOk9hgZ0l5v/n79K3e166seanmDt3AVUpl2l4zcbHde3iP3O658XSP79yQyy2XJAXhNJIYHhFw1ngYu4mvT+IVDsw3Xl/akkERZRytNXuu76FRiUYOjf/iLDMPzbRbX/vMWrTWFMtTDK9cXgBcvneZz/Z8RrNSzRIkhewmv2d+V4cgchhJDI/YYUkM7WoUS7jRbDKGvoAsOfRFwJUAlgcu56u2XyXY9s/Vf3h146vW9SG1hvBWk7ecEkf/Nf05dvsYFQpU4OJd48n1aoWrMaP9DIrnKW5tPvq/Nv9H3aJ1E/1mXC5/OWZ0mOGU+ITI6eQewyM6f72dyJhY/n7H5v5CyBmY1R5sR0TMIl1UI2Mi8V/sT4w5xq58Xpd51Clah8k7J7Pq3Kok93+x9ovMPT43VX3rk7Ll0hb+s/U/DtfPrE08QmR1Kd1jkDmfbUTHmjh94x5F8ua237D8Jfuk8H7y3QYzi1hzLM0WNkuQFACGrBtCo18aJUgKA2sOpFXpVtb1ucfnAkbPoLj5bEdsGpHs+5rMJl7e8DJXIq5YywavHWyXFGyHYm5ZOuGYQjv770xQJoTIGNKUZGPVoasAtK1u04xkioFrh+PXRx8F98z7Y+v7e19Ohp5MdNuLdV7klbqv0GJRiwTbZneabR1bB5Ifoz7gSkCC6Ri9c3mzsc9GXt34KsdvHwegy7Iuie3OV22/4skKTzL1Cfs5cn8/9zvvBrxLwPMBMo6OEC4kTUk24uZfODG5M3k8LR/+GyfCjmnw3Dyo1Qsy2RR8cU6HnqbP730S3bZn4B678e5tu0E+ui0pay+sZdbRWYQ+CE3xQaukTG83nXblU5gJTwjhdNJdNRWuhj+gRsn88UkBjKQAxoNsGZwUHsQ+IJfKleQoivdj7uPh5kGsOTbJpLCw28IEH/xp6aLZtWJXulaMH4/nTtQdQh6E8OzqZxOt/3vv3wmNCqVk3pLMPjqb91u8n6r3E0K4jiQGC601fwfe4slaNkMqR1qeqK3WBTzTPvRwWuy7vo8X178IGB+ycSN8NizeMMGQCba29t36WDM7OaqwV2EKexW2SzDzjs+jRN4SdPE1mpDi5hyQpCBE1iI3ny3OhUQAULKA0U8esxk+t4yTVCOJgfTSYF3QOurOq8v7O97HZDZh1mZCIkM4HXoarTWx5lgWn1psTQqA3bDPySWFTX02ZUhSSMqQ2kOsSUEIkXXJFYPFgYthAAxt5WsU/DM9fmP9pMeqSY2tl7by9l9vA8aY/SlN5uKu3DFpU4Jy/zL+lMhTgsjYSAp4FmDJ6SXM7jQ70QlkhBAitSQxWLyz7AgAFYvkNa4WNk00NkwIATf3NB3TZDbh7uZOjCmGgCsBCQZ2S06zks34b+f/orXmbNhZqhaummTdCc0dn2dXCCFSIonhEW5uyhg9FaBYzTRNvvPRzo/49cyviW4rnqc4m5/bbNcddP+g/Xi6e3I14ir7buxj66Wt1qeTlVLJJgUhhEhvkhiAm3ejAHi9nWXY4oWWsXeGrU1Q16zNzD8+n9pFa/N/+/6PtuXa8lq916zb99/Yn2RSAPjzmT8BKJm3ZIKeQaXzlaZnvp70rNzzcU5HCCEeiyQGYOPJGwA0qlAYgm2ei/A2Zs7aELSBN/96M9F9j98+zsxDM3mq0lP0qNSDVzcZ4w31qtyLdUHrKJGnBCt7r5QZs4QQWYYkBuDm3WiUghaVi8DMDkZhu/cAuBJxJcmkYOv387/z+/nfresf+3/Mx/4fOyVeIYRwJumuChy/epdyhfPglcsNwi4ZhW3eARIO6+Bfxp+A5wM4OuQoR4ccpVvFbgmOt2vALqfHLIQQzpLjrxhiTWZ2nrtlTON5brPdtlsPblmXf3ryJ1qUTjjG0GetP+Oz1p8BRpOTh5sHeT0y9mE4IYRITzk+Mfx99hb3H5qMZqQNllFD6w/i8r3LdFtuXA1Mazct0aTwqE6+nZwZqhBCZIgc35R07qbxxHOb6sXg5gmjsPdMa1IAaFdOBn4TQuQcOT4xHL96l2L5c1P8b8t4PhXbEBIZYt2+Z+Ae3FSO/zEJIXKQHN+UFHjzHrVKFYBTxvMFD5/7H+1/awPAe83ec2hIaiGEyE5y9FdhrTUXQu5Tq5CJ6HvBnGz+Ml3/eM66vV/1fi6MTgghXCNHXzGERERz/6GJV879i+YVyhF7Y71126Y+m1CZdFIeIYRwphx9xXDu5n0AFniGEmuTBL5t/62MVCqEyLFy9BXD2ZAIKrld5MdCBQCY0GwC/WpI85EQImfL4VcMEfgVjh8oT5KCEELk8MRw4UYY4flPkcds5uALB10djhBCZAo5OjHkvnGAY7k9qY83udxydKuaEEJY5djEcC8qhneZRJSbG60bvOzqcIQQItPIsV+TTwbf5mhe4+G15r4dXRyNEEJkHjn2iiH4zEECPY1pOysWrOjiaIQQIvPIsYnh7tUt/JHPGB5bxkISQoh4OfYT0e3ubwC83/x9F0cihBCZS6ZLDEqpLkqp00qps0qpcc56n9kF8wDwXLXnUqgphBA5S6ZKDEopd2Am0BWoBfRXStVK7/eJfhhNaC53CmpPGQ9JCCEekakSA9AUOKu1Pq+1fggsBnql95vsObwKgL5eTdP70EIIkeVltsRQBrhssx5sKbNSSr2ilNqnlNoXEhJCWnjlLcNzEXloUndQ2iMVQohsKrM9x5BYu462W9H6J+AngMaNG+tE6qeoSa1WNKm1Oy27CiFEtpfZrhiCgXI262WBqy6KRQghcqTMlhj2AlWVUhWVUp7A88BqF8ckhBA5SqZqStJaxyqlXgfWA+7AHK31cReHJYQQOUqmSgwAWus/gT9dHYcQQuRUma0pSQghhItJYhBCCGFHEoMQQgg7khiEEELYUVqn6RmxTEEpFQJcfIxDFAVupVM4WUFOO1+Qc84p5JxTp4LWulhSG7N0YnhcSql9WuvGro4jo+S08wU555xCzjl9SVOSEEIIO5IYhBBC2MnpieEnVweQwXLa+YKcc04h55yOcvQ9BiGEEAnl9CsGIYQQj5DEIIQQwk6OTAxKqS5KqdNKqbNKqXGujietlFLllFJblVInlVLHlVL/sZT7KKU2KqUCLa+FbfYZbznv00qpzjbljZRSRy3bpqtMPhm2UspdKXVQKbXGsp6tz1kpVUgptVQpdcry/90iB5zzGMvv9TGl1CKllFd2O2el1Byl1E2l1DGbsnQ7R6VUbqXUEkv5bqWUr0OBaa1z1D+M4bzPAZUAT+AwUMvVcaXxXEoBDS3L+YEzQC3gc2CcpXwc8JlluZblfHMDFS0/B3fLtj1AC4xZ9NYCXV19fimc+xvAQmCNZT1bnzMwD3jJsuwJFMrO54wxpe8FwNuy/iswNLudM9AaaAgcsylLt3ME/gX8YFl+HljiUFyu/sG44D+iBbDeZn08MN7VcaXTua0CngROA6UsZaWA04mdK8a8Fy0sdU7ZlPcHfnT1+SRznmWBzUB74hNDtj1noIDlQ1I9Up6dzzlu/ncfjOkB1gCdsuM5A76PJIZ0O8e4OpblXBhPSquUYsqJTUlxv3Bxgi1lWZrlErEBsBsoobW+BmB5LW6pltS5l7EsP1qeWU0D3gHMNmXZ+ZwrASHAXEvz2X+VUnnJxuestb4CfAlcAq4B4VrrDWTjc7aRnudo3UdrHQuEA0VSCiAnJobE2hezdJ9dpVQ+YBkwWmt9N7mqiZTpZMozHaVUD+Cm1nq/o7skUpalzhnjm15D4HutdQPgPkYTQ1Ky/Dlb2tV7YTSZlAbyKqUGJbdLImVZ6pwdkJZzTNP558TEEAyUs1kvC1x1USyPTSnlgZEUFmitl1uKbyilSlm2lwJuWsqTOvdgy/Kj5ZlRK6CnUioIWAy0V0r9QvY+52AgWGu927K+FCNRZOdz7ghc0FqHaK1jgOVAS7L3OcdJz3O07qOUygUUBEJTCiAnJoa9QFWlVEWllCfGDZnVLo4pTSw9D2YDJ7XWX9lsWg0MsSwPwbj3EFf+vKWnQkWgKrDHcrl6TynV3HLMwTb7ZCpa6/Fa67Jaa1+M/7stWutBZO9zvg5cVkpVtxR1AE6Qjc8ZowmpuVIqjyXWDsBJsvc5x0nPc7Q9Vh+Mv5eUr5hcfePFRTd7umH04DkHvOfqeB7jPPwxLguPAIcs/7phtCFuBgItrz42+7xnOe/T2PTOABoDxyzbZuDADSpX/wPaEn/zOVufM1Af2Gf5v14JFM4B5/whcMoS788YvXGy1TkDizDuocRgfLsfnp7nCHgBvwFnMXouVXIkLhkSQwghhJ2c2JQkhBAiGZIYhBBC2JHEIIQQwo4kBiGEEHYkMQghhLAjiUGINFJKjVZK5XF1HEKkN+muKkQaWZ6+bqy1vuXqWIRIT3LFIIQDlFJ5lVJ/KKUOW+YHmIgxhs9WpdRWS51OSqmdSqkDSqnfLGNYoZQKUkp9ppTaY/lXxVL+nOVYh5VS2113dkLYk8QghGO6AFe11vW01nUwRni9CrTTWrdTShUFJgAdtdYNMZ5SfsNm/7ta66YYT6VOs5R9AHTWWtcDembMaQiRMkkMQjjmKNDR8s3/Ca11+CPbm2NMpLJDKXUIY3yaCjbbF9m8trAs7wD+p5R6GWMCKSEyhVyuDkCIrEBrfUYp1QhjLKpPlVIbHqmigI1a6/5JHeLRZa31a0qpZkB34JBSqr7W+nZ6xy5EaskVgxAOUEqVBiK11r9gTCDTELiHMaUqwC6glc39gzxKqWo2h+hn87rTUqey1nq31voDjJm1bIdUFsJl5IpBCMfUBb5QSpkxRsIcgdEktFYpdc1yn2EosEgplduyzwSMUXwBciuldmN8GYu7qvhCKVUV42pjM8Z8vkK4nHRXFcLJpFuryGqkKUkIIYQduWIQQghhR64YhBBC2JHEIIQQwo4kBiGEEHYkMQghhLAjiUEIIYSd/wc2500Jio43ogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_actions = 5\n",
    "epsilon = 0.01\n",
    "\n",
    "agents = {\n",
    "    #\"Random\" : RandomAgent(n_actions), \n",
    "    \"EpsilonGreedy(eps=0.01)\" : EGreedyAgent(n_actions, epsilon),\n",
    "    \"UCB\": UCBAgent(n_actions),\n",
    "    \"Thompson\": ThompsonAgent(n_actions) }\n",
    "\n",
    "env = Bandit(n_actions)\n",
    "scores = {}\n",
    "\n",
    "for name, agent in agents.items():\n",
    "    score = get_regret(env, agent)\n",
    "    scores[name] = score\n",
    "\n",
    "plot_regret(agents, scores)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
